execve() (CONF_fork) launches the target as a fresh OSv application in its own
ELF namespace via application::run(new_program=true) and, matching Linux, does
not return to the caller.  A successful exec DID launch the program (the
elf::program new-namespace construction path is fine), but after the exec'd
program finished the unikernel would hang at shutdown instead of powering off:
the loader's application::join() blocked forever on the top-level app's
_terminated flag.

Root cause is the fork child thread's lifecycle, not execve or elf::program.
fork_thread() (arch/{x64,aarch64}/fork.cc) creates the child as a normal
*attached* sched::thread.  At construction the thread captures a shared_ptr to
the current application's application_runtime (sched.cc sets _app_runtime =
app->runtime()).  Nothing ever join()s the fork child -- the parent reaps it
through the fork pid registry / waitpid(), not sched::thread::join() -- so the
thread object is never destroyed and its _app_runtime shared_ptr is never
released.  With that reference outstanding the application_runtime's use count
never reaches zero, ~application_runtime never runs, the app's _terminated is
never set, and application::join() waits forever.  (This bit every fork(), and
was most visible after fork()+execve() where the child adopts the caller app's
runtime.)

Fix: create the fork child detached and dispose it in its cleanup.
- arch/{x64,aarch64}/fork.cc: mark the child attr().detached(), so on
  completion it is handed to the thread reaper (which runs its set_cleanup()),
  rather than sitting forever waiting to be joined.
- libc/process/fork.cc: the child's cleanup now also calls
  sched::thread::dispose(child) after the existing bookkeeping (record exit
  status if it fell off the end, free the copied user stack).  Disposing the
  thread releases its _app_runtime reference, letting the owning application's
  runtime drop to zero so join() completes and OSv powers off.  This mirrors
  the default detached-thread cleanup ([this]{ dispose(this); }).

Both files are compiled only under CONF_fork, so default OSv builds are
unchanged.

Tests: tst-fork test 2 now execs a real payload (/tests/payload-exit7.so, which
prints a marker and exit(7)s) instead of the previous _exit(7) sentinel that
masked whether execve actually launched anything; a return from execve() is now
treated as a failure.  Added tst-execve.so (fork+execve launches the payload and
its exit code is reaped; missing path returns -1/ENOENT) and payload-exit7.so.

Validated on x86-64 (KVM disk boot): tst-fork 10/10 and tst-execve 3/3 pass with
the real exec payload and OSv shuts down cleanly (5/5 repeat runs, no hang).

Copyright (C) 2026 Greg Burd

Follow-up to the base fork() PR: gives a forked child its own address space
with copy-on-write of private mappings, so a forked child has real memory
isolation like Linux fork() while MAP_SHARED / shm stays shared.  Also gated
behind CONFIG_fork (default off): with it disabled none of this is compiled and
OSv's single-address-space model and context-switch path are unchanged.

With CONFIG_fork enabled:
- mmu::address_space object (page-table root + vma_list); the previous global
  becomes 'address space 0' (kernel + init app).
- Per-thread current address space; the context switch loads the target CR3
  only when the address space differs (a no-op for AS0-only workloads).  Kernel
  PML4 entries are shared across all address spaces so OSv code + the kernel
  heap work identically after a switch.
- fork() clones the parent's vmas into a child address space: PRIVATE writable
  mappings are write-protected in both and copied on the first write (reusing
  OSv's existing COW fault machinery); MAP_SHARED / shm map the same physical
  pages (truly shared).  execve() returns the thread to AS0.
- Lock/condvar wait_records for fork-child (non-AS0) threads are allocated from
  the AS-shared kernel heap instead of the thread stack, so a wait_record queued
  on a shared kernel mutex resolves to the same physical page for any waker
  across address spaces (kernel-stack coherence).

Validated: tst-fork-cow proves a forked child's private memory stays private
while MAP_SHARED stays shared; tst-fork stays 10/10.

Known limitation (documented): the child's stack is still relocated+copied
rather than same-VA COW, so a child that unwinds a very deep call chain (e.g. a
multi-process server forking backends) can still hit a stack-fidelity issue
(tst-fork-deep); the same-VA stack fix is a further follow-up.  This PR delivers
memory-isolated fork for the common cases with COW proven.

Copyright (C) 2026 Greg Burd

Give a forked child its parent's EXACT user stack -- same virtual
addresses, private physical pages -- instead of a relocated copy with a
biased SP.  A relocated+biased stack leaves every saved rbp, return
address and &local in the copied frames pointing at the parent's stack
(off by the bias), so a deep unwind in the child dereferences the parent
and faults.  tst-fork-deep (fork 12 frames deep, child unwinds all the
way up and _exit()s a checked value) is the test that catches this; it
now passes.  Gated behind CONF_fork (default n); conf_fork=0 is unchanged
and builds/boots clean (fork code fully excluded, tst-mmap passes).

Four coupled changes make same-VA correct, layered on the Stage 2 COW
address space and the fork-child lifecycle fix (detached child +
dispose(child) in the reaper):

1. core/mmu.cc: clone_address_space() PRIVATIZES the forking thread's
   live stack VA range into the child AS -- fresh private pages that
   byte-copy the parent's stack, mapped writable (not COW: OSv runs
   kernel code with irqs off on the app stack, a COW fault there is
   illegal) at the SAME VA.  The parent's PTEs are untouched.

2. arch/x64/fork.cc + libc/process/fork.cc + include/osv/fork.hh: the
   child resumes with the caller's FULL callee-saved register context
   (rbx, rbp, r12-r15, rsp, rip), captured at fork() entry into
   osv::fork_resume_ctx and restored by the child trampoline off a
   scratch base register (so the restore never clobbers its own base).
   fork() jmps past its own epilogue, so without this the caller would
   resume with fork()'s internal register values -- a deep chain that
   keeps an accumulator in a callee-saved reg across fork() computes
   garbage.

3. arch/x64/arch-switch.hh: defer the fork CR3 switch to EXACTLY the
   rsp/rbp swap inside switch_to()'s asm.  Loading the incoming AS's
   CR3 early, while still executing on the outgoing thread's stack (the
   fpu-cw/mxcsr scratch saves), resolved those shared-VA stack writes
   through the wrong address space and clobbered a blocked thread's live
   stack (root-caused via hardware watchpoint: switch_to wrote garbage
   into a blocked parent's on-stack user_mutex slot in AS0).

4. core/lfmutex.cc + core/mmu.cc: a wait_record on a shared kernel
   condvar/mutex must live in the identity-mapped kernel heap (coherent
   in every AS) whenever a cross-address-space wake is possible.  Extend
   the Option-A rule from 'current thread is a fork child' to also
   cover 'any child address space is live' (live_child_address_spaces
   counter), so the parent in AS0 -- woken by a child that walks the
   queue through its own page tables -- is safe too.

libc/process/execve.cc: on a fork child, move the thread onto its own
kernel stack before switching to AS0 and running the target, so
reloading CR3 to AS0 cannot alias the same-VA app stack onto the
parent's physical page (which would corrupt a blocked parent).  The
child address space is destroyed HERE, on the kernel stack, exactly once:
the reap-time cleanup (libc/process/fork.cc) then sees the thread's AS is
AS0 (!= child_as) and skips its own destroy, so destroy_address_space()
runs once whether the child exits normally (reaper destroys) or execs
(execve destroys) -- no leak, no double-free.  A pre-exec existence check
keeps a failed exec returning to the caller.

Validated on x86-64 (KVM): tst-fork 10/10, tst-fork-cow 6/6,
tst-fork-deep 0 failures, tst-execve 3/3 with clean shutdown (no hang, no
double-free at reap), on smp=1 and smp=4, 5/5 repeat runs; conf_fork=0
builds and boots clean (fork excluded, tst-mmap passes).

…nZFS + patch series

Make the in-kernel ZFS implementation selectable at build time and integrate
OpenZFS 2.4.x without maintaining a fork, layered cleanly on the current
master (which already carries the reworked pagecache/ARC bridge, block
multiqueue, io_uring, musl 1.2.1, and modern-toolchain support).

conf_zfs switch (default bsd):
  - conf_zfs=bsd     - legacy in-tree BSD/Illumos ZFS (default, unchanged path)
  - conf_zfs=openzfs - vendored OpenZFS 2.4.x from external/openzfs
  Only one is compiled per build; both produce libsolaris.so plus the
  zpool/zfs userspace.  The shared solaris common objects (avl/nvpair/unicode/
  fm/list) are provided by OpenZFS in openzfs mode and by the in-tree copies in
  bsd mode.

Fork-free OpenZFS:
  - external/openzfs pinned to the PUBLISHED upstream tag zfs-2.4.2
    (github.com/openzfs/zfs @ 6330a45b), a plain submodule with no OSv fork.
  - The OSv platform-layer changes live as a 19-patch git series in
    modules/open_zfs/patches/ and are applied at build time by the Makefile
    (idempotent via the external/openzfs/.osv-patches-applied stamp) only in
    openzfs mode.
  - .gitignore keeps an explicit !modules/open_zfs/patches/*.patch exception so
    the series is tracked despite the global *.patch ignore.

ZFS-implementation-agnostic kernel:
  - cv_timedwait keeps the BSD relative-timeout semantics; a distinct
    openzfs_cv_timedwait (absolute deadline) is added in bsd/porting/netport1.cc
    and exported, so the same loader.elf serves both.  OpenZFS objects get
    -Dcv_timedwait=openzfs_cv_timedwait via OPENZFS_CFLAGS in openzfs_sources.mk.
  - core/pagecache.cc keeps master's live map_arc_buf/register_pagecache_arc_funs
    bridge (used by conf_zfs=bsd) and additively provides the three symbols the
    OpenZFS module needs (osv_free_pages, osv_pagecache_map_arc_page,
    osv_pagecache_register_arc_rele) via a separate borrowed-ARC-page path into
    read_cache.  The borrow class is renamed cached_page_arc_borrow to avoid
    colliding with master's arc_buf_t-based cached_page_arc.
  - drivers/virtio-blk.cc: give the completion thread a 256 KB stack so the ZFS
    vdev_disk_bio_done path does not overrun the default kernel stack.

Also: OpenZFS 2.4.x userspace lib/cmd build rules, openzfs_osv_compat.c, the
zfs-tools module.py (drops libzutil/libshare/libzfs_core/libtpool in bsd mode),
mkfs -m pool-root pinning gated on CONF_ZFS_OPENZFS, cpiod umount prefix
normalization so the host-side image build flushes the pool, and the ZFS
validation/benchmark test suite.

The conf_zfs=openzfs build linked the legacy BSD-ZFS kstat stub
(bsd/.../opensolaris_kstat.o) to satisfy kstat_create/install/delete,
but every OpenZFS module was compiled against the OSv SPL kstat_t in
external/openzfs/include/os/osv/spl/sys/kstat.h (~64 bytes: ks_data,
ks_ndata, ks_data_size, ks_flags, ks_update, ks_private, ks_private1,
ks_lock).  The BSD stub allocated only a 16-byte kstat_t, so OpenZFS
callers such as arc_init()/dnode_init() wrote ks_update/ks_private past
the end of the allocation, corrupting the malloc free list.  The next
kstat_create() then returned a garbage pointer and faulted at
ks_ndata = ndata (SIGSEGV -> abort with empty backtrace right after the
version banner, before mkfs ran).

Fix: implement OSv-native kstat_create/install/delete in
openzfs_osv_compat.c using the correct OpenZFS kstat_t layout (virtual
kstats; OSv has no /proc or sysctl consumer), and filter the BSD
opensolaris_kstat.o out of the openzfs solaris object list in the
Makefile so only the correct implementation is linked.

The zfs_builder guest now boots, runs mkfs (creates + mounts pool osv),
populates it via cpiod, and exports cleanly.

…mount

zfs_unmount() (called by zfs destroy, zpool export, and property
remounts) only unmounts a dataset if it can find it via
libzfs_mnttab_find() -> libzfs_mnttab_update() -> getmntent().  On OSv
that lookup went nowhere for two reasons:
 - MNTTAB pointed at a nonexistent path, so fopen(MNTTAB) failed before
   getmntany() ever ran; and
 - the getmntent/getmntany stubs in libzfs_util_os.c returned EOF.
Datasets auto-mounted by the kernel (zfs_domount at pool/dataset create
time, not through the libzfs do_mount path) were therefore invisible to
libzfs, the objset stayed owned, and zfs destroy / zpool export failed
with EBUSY / 'dataset is busy'.

Fix, as two edits to the OpenZFS patch series:
 - patch 0004: MNTTAB -> /etc/mnttab (created empty at boot, so fopen
   succeeds and libzfs proceeds to getmntany()).
 - patch 0020: add lib/libzfs/os/osv/libzfs_mnttab_os.cc implementing
   getmntent/getmntany over the live VFS mount table via
   osv::current_mounts() (filtered to ZFS), replacing the EOF stubs.

Wire the new C++ shim into the libzfs build, filtering the C-only
warning flags out of ozfs-cflags-common that a C++ translation unit
rejects.  Makes zfs destroy and zpool export/import work on OSv, which
the feature-coverage tests depend on.

zpool export/import intermittently aborted with

  Assertion failed: owner.load(...) == sched::thread::current()
  (core/lfmutex.cc: unlock: 221)

in condvar::wait().  Root cause: the OpenZFS userland thread pool
(lib/libtpool) hands jobs to worker pthreads that block in
pthread_cond_wait() on a shared tp_mutex.  On OSv a pthread mutex and
condvar ARE the kernel lockfree::mutex + condvar, whose wait-morphing
protocol transfers mutex ownership from the signalling thread to a
waiter.  During tpool_destroy() teardown that handoff races: a worker
returns from pthread_cond_wait() and re-enters it, unlocking a tp_mutex
it no longer owns -> lfmutex owner assertion.  Reproduced deterministically
on zpool import, whose device-scan (zutil_import.c) and mount
(libzfs_mount.c) thread pools default to hundreds of workers
(mount_tp_nthr = 512).

Fix: run tpool jobs synchronously on OSv (0021) and force serial dataset
mounting (0022).  OSv threads are cheap and these jobs are short, so
serial execution removes the worker/teardown machinery and the race at no
practical cost.  Verified 0/8 asserts (was 8/8) across create/export/
import/status cycles from a clean patch-series rebuild.

…dataset)

Patch 0017 reclaimed only the mount-root znode before dmu_objset_disown.
A mounted child dataset (e.g. pool/fs) leaves its own znodes live, each
holding an object bonus buffer that pins a dnode.  dnode_destroy never
runs, the objset os_dnodes list never empties, and spa_export() blocks
forever in spa_evicting_os_wait().  Reproduced deterministically by:
zpool create; zfs create pool/fs; zpool export (hang).

Walk z_all_znodes at unmount and inactivate every remaining znode, as
vop_inactive would, so the objset drains.  Verified 6/6 completions
(was 0/6, hung) for create+child-fs+export+import+destroy.

OSv read_partition_table() names MBR slots 0-based (first slot is
/dev/vblkN.0) and exposes a raw unpartitioned disk directly as /dev/vblkN
with no child node.  zfs_append_partition() unconditionally appended .1, so
\x27zpool create test /dev/vblk1\x27 on a raw wiped NVMe failed with
\x27cannot open /dev/vblk1.1: No such file or directory\x27.

Fix (patch 0023): only append .1 when that partition node actually exists;
otherwise use the whole raw disk as-is, matching Linux/FreeBSD whole-disk
behavior (ZFS writes its own labels to the whole device).

Also adds FINDINGS-osv-openzfs.md documenting Bug 1, Bug 2, and the
page-allocator-under-memory-pressure investigation (>=4G runtime guests).

Verified: zpool create on a raw disk now succeeds, pool ONLINE.

…r 0)

zpool list faulted with strlen(NULL) in print_line() because
zpool_prop_column_name() returned NULL for a default list column on OSv.
The pool data row is correct; only the header label resolves NULL.
Guard header against NULL, falling back to the ZFS missing-value glyph,
so the command cannot crash.

zfs_is_readonly() was hardcoded B_FALSE, so a readonly=on dataset stayed
writable on OSv.  Read the readonly property at mount into a new
zfsvfs->z_readonly, return it from zfs_is_readonly(), and reject the write
vnops (write/create/remove/rename/mkdir/rmdir/setattr/truncate/symlink)
with EROFS.  Verified writes fail EROFS after remount and readonly=off
restores writability.

vdev_disk_open() never set vd->vdev_has_trim, so zpool trim/autotrim
reported trim not supported.  Enable it optimistically (OSv has no
capability query): the ZIO_TYPE_TRIM->BIO_DISCARD path reclaims space on
virtio-blk with discard, and devices without discard get ENOTSUP mapped
gracefully.  Verified zpool trim completes 100% with discard=unmap.

…ryption/dedup/draid/TRIM/O_DIRECT/etc)

Head-to-head storage microbenchmark of the two ZFS ports on identical
a bare-metal host, 8 GiB KVM guest, single-vdev (raw NVMe) and raidz2
(7x20 GiB). No Postgres/HammerDB - isolates the filesystem layer.

Key finding (the ARC-bridge measure): mmap of a 512 MiB file costs BSD-ZFS
~5 MiB extra RAM vs OpenZFS ~501 MiB - BSD's unified ARC/page-cache bridge
shares pages, OpenZFS double-buffers via its borrow path. Same root cause
drives BSD's warm-read and cached-random-read wins. OpenZFS wins the
write/CPU paths (seq write +34%, lz4 +56%, metadata +63%, fsync +11%) and
uniquely offers O_DIRECT (ARC bypass, ~raw-ceiling, BSD has no direct=).

Harness (scripts/bench/): pure-C zfs-bench.c (a g++ .so pulls libstdc++
symbols OSv can't resolve at load; C avoids GLIBCXX/CXXABI/__isoc23). Runs
off the zfs_builder bootfs - no usr.img cpiod populate. rebuild-bench.sh,
run-bench.sh, run-all.sh drive it; results.tsv is the raw data.

Copyright (C) 2026 Greg Burd

…en_zfs/

Per review: external/ is reserved for things compiled into the kernel
(libfdt, acpica).  The vendored OpenZFS tree is a module dependency, so
move the submodule from external/openzfs to modules/open_zfs/openzfs
alongside its patch series and module.py.

Updates: .gitmodules path (still points at the published upstream
github.com/openzfs/zfs at tag zfs-2.4.2, no fork), the OZFS/OPENZFS make
variables (Makefile + bsd/sys/cddl/openzfs_sources.mk), the parse-time
patch-apply step (stamp path, -d guard, and the git -C ... apply path,
whose relative patch prefix changes from ../../modules/open_zfs/patches
to ../patches now that the submodule sits one level deeper), and the
doc comments that named the old path.

The per-child COW address space (clone_pt_level<1>) shared any 2 MB large
page verbatim between parent and child instead of copy-on-writing it:

    if (ppte.large()) { child_pt[i] = ppte; continue; } // 2MB: share as-is

so a forked child writing a private large mapping scribbled the parent(s)
too.  This hit large malloc (>= 2 MB, mmap-backed), large MAP_ANON regions,
and any 2 MB-backed private writable mapping in the application slots.

Fix: for a WRITABLE large PD entry, split it to 4 K in the parent in place
(the existing split_large_page primitive) and fall through to the normal 4 K
clone path, which already makes the correct per-page decision -- private
writable -> COW, MAP_SHARED -> stays shared, read-only -> shared as-is.  A
read-only large page can never diverge, so it is still shared verbatim.  This
reuses the whole 4 K COW fault machinery (clone_pt_level0 +
handle_cow_write_fault); the only cost is one 4 K page table per split large
page, acceptable under the opt-in CONF_fork.

Verified on x86-64 (KVM microvm):
  - a standalone musl-PIE fork test now isolates a private >2 MB malloc across
    fork (child write does not change the parent; parent write does not change
    the child), while a MAP_SHARED region stays mutually visible;
  - tst-fork 10/10, tst-fork-cow 6/6, tst-fork-deep and tst-execve all pass
    (no regressions);
  - conf_fork=0 builds clean with the whole fork path (this change included)
    compiled out (clone_address_space absent from loader.elf).

Note: this fixes large-page (application-slot) COW.  OSv's small-object
malloc heap lives in the shared kernel identity map (mem_area::mempool), which
fork() shares by design; isolating that is a separate change.

All gated behind CONF_fork (default off).

Copyright (C) 2026 Greg Burd

Stock fork-per-backend PostgreSQL corrupts memory on OSv because OSv's
small-object malloc heap lives in the kernel identity map (mem_area::mempool,
VA >= 0x400000000000), which is shared verbatim across every address space and
cannot be copy-on-write cloned on fork().  A forked child writing to a heap
object inherited from its parent scribbles the SHARED heap.

Give application allocations a home in an ordinary anonymous mmap region in the
app-slot VA range (below phys_mem), so clone_address_space()'s existing COW
machinery isolates the whole app heap per child.  virt_to_phys() already
page-walks addresses below phys_mem, so arena pages stay DMA-usable.

The arena is a segregated free-list whose bookkeeping lives entirely in kernel
BSS (never in arena pages), so managing it never faults an arena page and never
recurses into malloc during fork's own page-table work.  std_malloc routes an
app thread's allocations to the arena (decided by is_app at malloc time);
free()/realloc() dispatch purely by address range, so cross-thread alloc/free
is always handled by the allocator that owns the address.

All gated behind CONF_fork (default n): with fork disabled none of this is
compiled and the heap behaves exactly as before.

tst-pgfork.c proves a forked child's writes to inherited AND new small/large
heap objects do not leak into the parent.

Author: Greg Burd

Progress on the fork heap arena:
- arena allocator reworked to use a preemption-disabling spinlock (not a
  sleeping mutex, which is illegal when malloc runs preempt-disabled) with the
  page-touching header write done outside the lock.
- clone_address_space() pre-reserves the child address_space object and moves
  the child vma-list construction (anon_vma allocations) OUT of the vmas_mutex
  critical section, and snapshots parent vmas under the lock into a pre-grown
  vector -- avoiding malloc under the fork lock (the malloc-during-fork trap).
- sched::thread objects and coherent wait_records forced to the identity kernel
  heap (kernel_heap_scope) so cross-address-space kernel structures are not
  COW-private.

Known blocker (documented in /tmp/fork-arena-result.txt): with the arena active,
the first fork() still hits an illegal page fault (page_fault: preemptable()
assert) during/around clone -- a COW write fault occurring in a
preemption/irq-disabled window.  Arena allocator validated standalone
(tst-realloc passes).  conf_fork=0 verified clean (no fork_arena symbols, boots).

Author: Greg Burd

… the fork lock

Further hardening of the fork heap arena (still gated on CONF_fork, default n):
- arena allocator is now fully lock-free (Treiber-stack free lists + atomic bump
  pointer): no preemption-disabling lock is held while touching a chunk, so a
  post-fork copy-on-write page fault on a chunk is always serviceable (OSv
  forbids faulting with preemption/interrupts disabled).
- clone_address_space(): defer flush_tlb_all() and the live_child_address_spaces
  bump until AFTER releasing vmas_mutex, and build the child vma_list from a
  snapshot outside the lock -- a write that COW-faults a kernel .bss global (or
  any malloc) must not run while vmas_mutex is held for write, or the COW fault
  handler self-deadlocks re-acquiring it.

Remaining blocker: COW-cloning the arena subtree specifically still triggers an
illegal (preemption-disabled / recursive) page fault at the first fork();
sharing the arena instead lets fork() run (5/5 early tst-fork checks) but loses
heap isolation.  Documented in /tmp/fork-arena-result.txt.

Author: Greg Burd

A thread's TLS block and tiny/large syscall stacks hold state the kernel reads
and WRITES from preemption/interrupt-disabled contexts (TLS __thread vars; the
kernel runs syscalls on the syscall stack).  If they were allocated from the
COW fork arena (an app thread creating a thread after the arena is live), fork
would write-protect them and the next preempt-disabled write would take an
illegal COW page fault.  Force these allocations to the identity kernel heap
via kernel_heap_scope, matching the thread-object and wait_record treatment.

Author: Greg Burd

…cycle + execve-continuation allocs to identity heap

Root cause (found via qemu+gdb hardware watchpoint): the kernel_heap_scope
guard around aligned_alloc in setup_tcb was being ELIDED by the compiler.
force_kernel_heap was a plain __thread unsigned; GCC models aligned_alloc (and
operator new) as not reading global memory, so the paired ++/-- around a single
such call was dead-code-eliminated -> app-thread TLS/thread objects landed in
the COW arena -> writing preempt_counter (in a COW arena page) during fork
triple-faulted.

Fixes (all gated CONF_fork):
- force_kernel_heap -> volatile, so the scope's inc/dec survive around
  builtin-modeled allocators (verified in disassembly + a minimal repro).
- wrap the whole sched::thread construction in make() in a kernel_heap_scope so
  the thread object, _detached_state, and _wakeup_link._helper (all touched by
  the scheduler cross-AS with preemption off) go to the identity heap.
- __cxa_thread_atexit_impl linked_destructor nodes -> identity heap.
- execve() continuation strings (s_path/s_args/s_env): allocate their backing
  storage under kernel_heap_scope so it survives destroy_address_space() of the
  child COW AS (was reading a freed arena buffer -> 'executable too short /').

Validated: tst-pgfork PASS 9/9 (child small-heap writes do NOT leak to parent -
the arena's core purpose works). tst-fork now passes 4/10; fork+execve (test 2)
still hits a nested page-fault under investigation.

…hes the arena

A forked child's fork+execve (tst-fork test 2) aborted with
'exception_depth <= 1' during ELF demand-paging.  The exec'd program's
file-backed pages fault in through vm_fault -> pagecache/ROFS, which
allocates cached_page bookkeeping.  On an app thread those allocations
route to the COW fork arena; first-touching a fresh arena page WHILE
servicing a fault (already holding vma_list_mutex, possibly one exception
deep) takes a second page fault, exceeding exception_depth and asserting.

vm_fault services KERNEL work: pagecache pages and filesystem demand-paging
buffers are shared kernel infrastructure that must be identity-mapped and
never a COW arena page.  Wrap the whole handler in a kernel_heap_scope
(gated CONF_fork) so no arena page is allocated or first-touched during
fault servicing.  Restores tst-fork 10/10 with the arena enabled.

…loc)

The detached-thread reaper kept its pending zombies in a std::list<thread*>,
whose push_back() in add_zombie() allocates a list node.  Two problems with
that under the fork arena:

  1. add_zombie() runs from a terminating thread.  For a fork child that is an
     APPLICATION thread executing in the child's COW address space, so the
     std::list node is malloc'd into the fork arena -- landing on a
     COW-private page at an arena VA.  The reaper drains the list from AS0,
     where that same arena VA resolves to a DIFFERENT physical page: it reads
     a garbage 'thread*' and join()s a bogus pointer forever.  OSv then never
     runs the child's cleanup, the child's application_runtime reference never
     drops, application::join()'s wait_until(_terminated) never completes, and
     the guest hangs at shutdown instead of powering off.

  2. The node allocation can also fault a fresh heap page from a
     preemption-disabled termination window.

Switch _zombies to a boost::intrusive::list threaded through a new
thread::_zombie_link hook.  The link lives inside the thread object (identity
kernel heap), so add_zombie() performs NO allocation: coherent across address
spaces and safe from a non-preemptable context.

… heap

async::run_later()/timer task nodes (one_shot_task, percpu_timer_task) were
allocated from the COW fork arena. The async worker walks its intrusive task
list with preempt_lock held (preemption disabled); touching a COW arena page
there faults, and OSv forbids page faults in non-preemptable context
(assert(sched::preemptable()) in page_fault). Force these nodes onto the
identity kernel heap via fork_arena::kernel_heap_scope, matching the existing
wait_record / child-registry kernel-heap pattern.

Surfaced by booting stock musl PostgreSQL 18 with CONF_fork: the postmaster's
first forked child (checkpointer) tripped this the moment the async worker
fired a TCP-stack run_later callback. Clears that wall; shared-memory attach
across fork() then works (verified: child reads PG ProcGlobal shared state).

fork_arena::init() reserved the 512 MiB arena VA with map_anon(mmap_fixed)
WITHOUT populating it, so pages faulted in lazily on first touch.  Under real
concurrent PostgreSQL load fork_arena::alloc() gets entered from an IRQs-off /
preemption-off context (mid-exception, kernel work), and first-touching a
freshly bump-carved page there DEMAND-FAULTS -- which page_fault forbids
(assert(ef->rflags & rflags_if) / assert(preemptable)) -> abort.

Add mmu::mmap_populate to the map_anon flags: all 512 MiB is RAM-backed at
init, so alloc's first write always hits a present page and can never fault.
That makes alloc safe to call from ANY context.  clone_address_space() still
COW-clones the whole vma per child; the child only WRITE-faults (COW break)
from app context with irqs/preemption on, so the original fault-context
invariant is preserved on the child path.

512 MiB fully backed is acceptable for a per-app fork heap; boot is unaffected
(~3-4 s) and the whole fork suite stays green (tst-fork 10/10 x5, tst-pgfork
9/9, tst-fork-cow/deep/execve pass).  All gated CONF_fork; conf_fork=0
unchanged (arena compiles to an empty TU, exits 0).

Also bump apps to the pg18-fork test module (musl PG18 fork-per-backend image).

…tion bug

A fork child that spins deep on its same-VA stack while being heavily
preempted (touching .data/.bss globals, its own stack, FPU/SSE, and taking
syscalls + COW faults) has its register context corrupted on the preemptive
context-switch resume path -- a callee-saved register (rbp) is reconstructed
from a .text immediate and a pointer resolves into the same-VA stack, giving a
page fault outside the application.  Deterministic on -smp 1 and -smp 2.

The existing fork tests only fork short-lived children preempted a handful of
times, so they never exercised this window.  The manifest entry is auto-derived
from the tests list; the test itself is unconditional (fork() just fails when
built conf_fork=0).

Author: Greg Burd

…eempt test + mmap-in-child repro

Three related changes from investigating the PostgreSQL fork-child wall:

1. libc/signal.cc: the global "waiters" std::list (one per signal, populated by
   wait_for_signal / drained by unwait_for_signal) is manipulated cross-address-
   space -- a fork child and its parent both push/remove on the same shared
   list.  Its list nodes were allocated from the COW fork arena, so a child's
   node became a COW-private page whose next/prev links were inconsistent with
   the shared sentinel seen from the other AS; a later list::remove() traversal
   (reached via sigprocmask in the checkpointer child) followed a dangling link
   and page-faulted with interrupts disabled -> assert(rflags & IF) in
   page_fault.  Force the identity kernel heap for those node alloc/free, the
   same rule already applied to thread objects and wait_records.  Proven with
   gdb: this was one of the two PG crash variants after "InitAux shmem read OK";
   the fix eliminates it.  Gated CONF_fork.

2. tests/tst-fork-preempt.cc: reframed honestly.  A long-lived fork child spun
   deep on its same-VA stack while heavily preempted (compute + .data/.bss
   globals + stack + FPU + syscalls) PASSES -- proving the preemptive context
   switch itself preserves a fork child's register/stack context (this refutes
   the earlier hypothesis that the deferred-CR3 same-VA switch was the wall; see
   report).  Green on -smp 1 and -smp 2.

3. tests/tst-fork-child-mmap.cc: standalone repro (NOT in the suite -- it aborts
   the unikernel) of a separately-found real bug: a fork child's own mmap() then
   write SIGSEGVs, because only the page-FAULT path is per-child-AS aware while
   the mmap/munmap ALLOCATION path (mmu::allocate / map_anon via the global
   vma_list + vma_range_set) is not.

See /tmp/pg-preempt-fix.txt for the full analysis.

Author: Greg Burd

…or semaphores, waitqueues and epoll watcher lists -> stock PostgreSQL survives a heavy pgbench -i bulk load on OSv

The storage / shared-buffer coherence wall under `pgbench -i` (parallel index
build / checkpoint large write) was a ramfs segment DATA BUFFER landing in one
forked backend's private COW address space:

  ramfs_enlarge_data_buffer() grows a file's segment with malloc().  Once the
  segment exceeds the fork arena's 2 MiB max_alloc it falls through to
  malloc_large()'s size>=huge_page && !contiguous branch, which reserves an
  APP-SLOT (VA < 0x400000000000) anonymous mmap in the CURRENT process's address
  space.  With per-child COW address spaces that buffer exists ONLY in the
  backend that enlarged the file; a sibling backend read()ing/write()ing the
  same ramfs file then memcpy's (uiomove) against an app-slot VA unmapped in its
  own page tables:

    page fault outside application, addr: 0x2000....
    RIP: memcpy_repmov_ssse3 ; pread/pwrite -> vfs_file::read/write -> ramfs

  i.e. the exact fault under `pgbench -i -s 5`.  A kernel_heap_scope does not
  help large allocations: it only skips the fork arena; malloc_large still uses
  app-slot map_anon for huge non-contiguous requests.

Fix (fs/ramfs/ramfs_vnops.cc): allocate the segment data buffer as physically-
contiguous IDENTITY-mapped memory (memory::alloc_phys_contiguous_aligned, VA >=
0x400000000000, in the kernel PML4 slots clone_address_space() shares verbatim),
so every address space reaches the same file image.  Plain free() dispatches by
address to free_large (identity path), so the existing free(data) sites are
correct from any address space.

Investigating the concurrent-writer path uncovered three more instances of the
same "kernel structure a cross-AS waker dereferences must be identity-mapped"
rule (all #if CONF_fork; non-fork byte-identical):

- libc/sem.cc: a pshared POSIX semaphore's posix_semaphore object (its _val,
  _mtx and _waiters list) was heap-allocated on the creating app thread -> COW
  arena.  A waiter (sem_wait) blocked on its private _waiters and a poster
  (sem_post) in another backend bumped a private _val -> lost wakeup, and
  post_unlocked() GP-faulted dereferencing a stack wait_record cross-AS.  Force
  the object onto the identity kernel heap.

- core/semaphore.cc: the semaphore wait_record itself is queued on the shared
  _waiters list and read by a cross-AS poster.  Route it through the same
  fork_child_needs_heap_wait_record() gate lockfree::mutex uses (identity heap
  when a cross-AS wake is possible; on-stack fast path in AS0).

- include/osv/waitqueue.hh + core/waitqueue.cc: wait_object<waitqueue> queued a
  stack wait_record on the shared _waiters_fifo (epoll's activity waiters, etc.)
  that a cross-AS waker walks.  Give it a coherent_wait_record, like condvar.

- fs/vfs/kern_descrip.cc: file::epoll_add allocated f_epolls (the per-file list
  of watching epoll instances) on the registering backend's COW arena; the
  RX/netisr thread walks so->fp->f_epolls cross-AS to wake watchers.  Keep the
  vector and its growth on the identity kernel heap.

Regression test tests/tst-fork-shared-write.cc (registered in
modules/tests/Makefile): a forked child ftruncate()s + writes a large pattern to
a ramfs file (forcing the >2 MiB segment buffer into its AS), then a SIBLING
forked child and the parent read it back.  Reproduces the "page fault outside
application" memcpy abort on HEAD; passes with the fix.

Results (KVM -m 4G -smp 2, pg18-fork, ramfs /data):
  pgbench -i -s 5 (heavy bulk load): completes -- 500k rows, vacuum, primary keys.
  pgbench -c 1 -T 20 (sustained heavy TPC-B write): 41,042 txns, 2053 tps, 0 failed.
  pgbench -S -c 4 (concurrent read): sustains ~45-50k tps for a bounded window.
  All fork regression tests: 0 failures (tst-fork, -cow, -deep, -serial,
  -arena-freelist, -preempt, -irq-cow, -conn-socket, -socket, -pgfork, -execve,
  tst-fork-shared-write NEW).  conf_fork=0 builds clean (CONF_fork==0).

HONEST STATUS: the storage/shared-buffer coherence wall this targeted is FIXED
(pgbench -i completes; sustained single-writer load completes).  Sustained
CONCURRENT load still hits two distinct, precisely-symbolized cross-child-COW
walls that are NOT storage coherence: (1) a post-fork MAP_SHARED write-visibility
race (pre-existing tst-fork-posix-shm part 3, fails on clean HEAD, passes only
when gdb/instrumentation serializes the CPUs), and (2) a park_timers boost
intrusive-list double-insert assert in sched::cpu::reschedule_from_interrupt
during epoll_file::wait under concurrency.  Both are in the #if CONF_fork
scheduler/timer/AS-clone machinery, the next investigation for a full HammerDB run.

PR cloudius-systems#1456 relevance: MAP_SHARED / cross-AS coherence across per-child COW address
spaces is core to it; these are five more instances of the shipped rule (struct
file, signal table, thread objects, arena free-list, DSM registry, mbufs, app
runtime, net_req): any kernel structure a forked child or a cross-AS waker reads
must live on the identity kernel heap, never the COW fork arena.

…, per-backend pid + SIGURG latch routing, and park_timers migration -> stock PostgreSQL sustains CONCURRENT multi-client load on OSv

Sustained concurrent pgbench now completes with 0 failed transactions
(pgbench -c 4/-c 8 read-write, -S -c 8 read-only) -- the gate to HammerDB.
Four cross-AS coherence walls fixed, all #if CONF_fork (non-fork byte-identical):

(A) epoll_file map/_activity node allocations landed in the COW fork arena and
    were freed cross-AS by the RCU quiescent-state thread (AS0) -> fork_arena
    chunk-magic abort under sustained epoll load.  Force them onto the identity
    kernel heap (core/epoll.cc EPOLL_KH).

(B) rcu_defer's on-stack wait_record, linked into *percpu_waiting_defers, was
    walked cross-AS by the RCU thread with preemption disabled -> page fault in
    non-preemptable context (exception_depth assert) for a fork-child caller.
    Use coherent_wait_record (identity heap under fork) (core/rcu.cc).

(C) OSv reported one pid (OSV_PID) for every forked backend, so PostgreSQL's
    SetLatch saw owner_pid == MyProcPid for cross-backend latches and took the
    LOCAL self-pipe path -> the waiting backend was never woken and a
    heavyweight-lock wait wedged forever (0 tps, 0 failed).  getpid() now
    reports each fork child's distinct pid (runtime.cc, libc/process/fork.cc
    pid<->AS registry), and kill() routes a fork-child-targeted signal handler
    into the target backend's address space so latch_sigurg_handler pokes the
    right self-pipe (libc/signal.cc).

(D) park_timers' per-CPU parked-threads list was not migration-coherent: a
    parked thread migrated (load_balance / thread::pin) while still linked on
    its source cpu tripped the intrusive-list double-insert.  Track the owning
    cpu (thread::_parked_cpu), unlink at every migration point, guard push_back
    with is_linked() (core/sched.cc, include/osv/sched.hh).  Corrects 1041518.

(E) /dev/urandom was unseeded at startup (yarrow accumulates entropy lazily),
    so an early read blocked/EINTR-looped and PostgreSQL backends failed with
    'could not generate random cancel key' (pre-existing, racy, not fork).  Read
    directly from RDRAND when the CSPRNG is unseeded (drivers/random.cc; not
    fork-gated).

New regression tests (tst-fork-timer-park-stress reproduces A+B;
tst-fork-shared-race the post-fork MAP_SHARED path; tst-fork-urandom E +
per-child pid).  Full fork suite 0 failures; conf_fork=0 compiles clean +
non-fork sched/epoll/mmap green.  tst-fork-posix-shm part 3 (WALL 1, pre-existing
one-shot MAP_SHARED visibility race) still intermittent; does not block
concurrent pgbench.

… zfs

Restructure the in-kernel ZFS build into the module layout requested in
PR cloudius-systems#1423 review (issue cloudius-systems#1201), mirroring how java is provided by
openjdk*-from-host:

  - modules/zfs        placeholder; module.py selects the impl by conf_zfs
                       (openzfs -> open_zfs, else -> bsd_zfs). Still ships the
                       shared libsolaris.so via usr.manifest.
  - modules/bsd_zfs    provides = ["zfs"]. Owns the legacy BSD/Illumos ZFS
                       build rules in bsd_zfs_sources.mk (the solaris compat
                       list, the zfs core list, and the conf_zfs=bsd
                       object/flag selection) moved out of the top-level
                       Makefile.
  - modules/open_zfs   provides = ["zfs"]. openzfs_sources.mk moved here as
                       open_zfs_sources.mk and extended with the
                       conf_zfs=openzfs object/flag selection (solaris +=
                       $(openzfs-all), kstat filter, OpenZFS CFLAGS).

The top-level Makefile now includes modules/bsd_zfs/bsd_zfs_sources.mk
unconditionally (defines the shared solaris/zfs lists) and, for
conf_zfs=openzfs, additionally includes modules/open_zfs/open_zfs_sources.mk.
The provider modules register the "zfs" capability during placeholder import,
exactly like the java placeholder, so no include collision occurs.

Build-structure refactor only; the same objects build for each mode. Kernel
impl-agnostic bits (openzfs_cv_timedwait, the -DCONF_ZFS_OPENZFS redirect, the
OpenZFS patch-apply) stay in the kernel/top-level Makefile.

Validated under KVM:
  conf_zfs=bsd     -> EXIT=0, cpiod finished, pool populated, libsolaris.so.

… pool at root

In open_zfs mode the ZFS pool was not usable as the root filesystem: the
guest mounted an empty pool at / and powered off with "Failed to load
object: /hello" (issue reported by wkozaczuk on PR cloudius-systems#1423).  bsd_zfs
worked end to end.

Root cause: the per-object flag that makes mkfs pick the OpenZFS
pool-root mountpoint convention,

    $(out)/tools/mkfs/mkfs.o: CXXFLAGS += -DCONF_ZFS_OPENZFS

was placed in the early conf_zfs=openzfs block, ABOVE the line where
`out` is defined (out = build/$(mode).$(arch)).  At that point
$(out) expands to the empty string, so the target-specific variable
bound to the target `/tools/mkfs/mkfs.o` instead of the real
`build/release.x64/tools/mkfs/mkfs.o`, and the define never reached
the compile.  mkfs.cc therefore always took its #else (BSD) branch:

    zpool create -f -R /zfs osv        # no -m /

With BSD ZFS the pool-root mountpoint defaults to / so the pool mounts at
the -R altroot /zfs, and cpiod --prefix /zfs/ populates the osv dataset;
the loader then mounts osv and finds /hello.  OpenZFS instead defaults the
pool-root mountpoint to /<pool> (/osv), so under -R /zfs the pool mounted
at /zfs/osv.  cpiods /zfs/... writes then landed on the builders ramfs
(nobody was mounted at /zfs) and were lost on shutdown, leaving the on-disk
osv dataset empty apart from the osv/zfs child mountpoint stub.  At boot the
mount + pivot both succeeded, but every lookup in the root ZAP returned
ENOENT, so /etc/fstab and /hello were invisible.

Fix: define a plain make variable (conf_zfs_openzfs) in the early block and
attach the -DCONF_ZFS_OPENZFS per-object flag AFTER `out` is defined,
gated on that variable.  mkfs.o now compiles with the define, so open_zfs
uses:

    zpool create -f -R /zfs -m / osv   # mountpoint / -> mounts at /zfs

and cpiod populates the osv dataset exactly as BSD does.  The pool now
mounts at / and the app runs.  Kernel stays impl-agnostic; the only
consumer of CONF_ZFS_OPENZFS is the userspace mkfs build tool, and the
change is gated on conf_zfs=openzfs so conf_zfs=bsd is byte-for-byte
unchanged.

Validated under KVM, both modes booting native-example
end to end from the ZFS root:
  conf_zfs=bsd      -> zfs: mounting osv from device /dev/vblk0.1; /hello runs
  conf_zfs=openzfs  -> ZFS: root mounted ok; /dev,/proc,/sys remount; /hello runs
Write+readback of a file at the ZFS root and its survival across a reboot
confirm the mount is a real, persistent ZFS root.

…fs=openzfs

The six ZFS tests that drive the OpenZFS userspace C API via
dlopen("libzfs.so") -- tst-zfs-recordsize, tst-zfs-direct-io,
tst-zfs-encryption, tst-zfs-db-sim, tst-zfs-trim and misc-zfs-mmap-bench --
plus the libzfs* user-space libraries they need, only exist in
conf_zfs=openzfs.  They were, however, added to the manifests for any fs=zfs
build, which broke conf_zfs=bsd two ways:

 1. usr.manifest emitted /usr/lib/libzfs_core.so (and libzutil/libshare/
    libtpool) gated on $(filter zfs,$(fs_type)).  conf_zfs=bsd does not build
    those four libraries, so the image build failed at upload time with
    "ERROR: file upload failed: ... No such file or directory:
    './libzfs_core.so'".

 2. The dlopen tests were in the always-on `tests` list, so a conf_zfs=bsd
    test.py run collected them from usr.manifest even though libzfs.so /
    libzfs_core.so are not present -- they could only ever print
    "SKIP: cannot load libzfs.so" (noise), and tst-zfs-trim in particular
    looked like it stalled.

Split those six tests into zfs-openzfs-only-tests and append them (and the
zfs-user-libs manifest emission) only when conf_zfs=openzfs, matching how the
top-level Makefile builds the libraries.  tst-zfs-crucible-stress, misc-zfs-io
and misc-zfs-arc do not dlopen libzfs and stay in both modes.

conf_zfs reaches this Makefile through the environment (scripts/build exports
every name=value build argument); default it to bsd here to match the
top-level Makefile when the module Makefile is invoked directly.

The system-lua/luarocks fallback added to modules/lua/Makefile is a build
convenience unrelated to the selectable BSD/OpenZFS ZFS work in this PR.
Restore the file to its upstream/master state so it is not carried by cloudius-systems#1423;
it can be proposed on its own.

…is busy")

Building a ZFS image ends with the zfs_builder running
  /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/;
  /zfs.so set compression=off osv; /zpool.so export osv
and printing "cannot export 'osv': pool is busy" at the very end.

mkfs auto-mounts the pool root (osv at /zfs) and the osv/zfs child
(at /zfs/zfs), and cpiod writes into them.  zpool export's own
zpool_disable_datasets() unmounts datasets it finds in /etc/mnttab, but OSv's
mount shim never populates mnttab, so it finds nothing to unmount; the kernel
then refuses the export with EBUSY because the still-mounted datasets hold the
objsets (spa refcount != 0).  The message is cosmetic -- the image is fully
written -- but it looks like a failure.

Drain the pool explicitly before exporting: unmount the datasets by mountpoint
(children first) with the OSv-native /tools/umount.so, which calls umount(2) ->
VFS_UNMOUNT -> the real zfs_umount -> dmu_objset_disown, dropping the spa
refcount to zero so the export succeeds cleanly.  This is mode-independent and
avoids the bsd "zfs unmount" command, which needs statfs2mnttab (absent from
the builder image).  umount.so is already built into tools; add it to the
zfs_builder bootfs so it is available there.

…irtio-blk req cross-AS coherence -> stock PostgreSQL on ZFS-on-NVMe forks its aux processes on OSv

Two cross-AS coherence fixes uncovered running stock PostgreSQL 18.4 with /data
on a ZFS pool (fs=zfs conf_zfs=openzfs conf_fork=1), which the prior ramfs image
never exercised:

1. libc/pthread.cc: the process-global atfork_handlers std::vector backing
   store landed in the COW fork arena (register_atfork runs on an app thread);
   a forked child read a DIVERGENT copy.  Additionally, apps run via osv::run()
   before the main app (zpool.so/zfs.so in the boot cmdline, import_extra_zfs_
   pools) register atfork handlers via __register_atfork and then EXIT, leaving
   dead handler pointers into unloaded objects.  When PostgreSQL then fork()s,
   __osv_run_atfork_prepare() called a garbage/dead pointer -> instruction-fetch
   SIGSEGV.  Fix: (a) allocate the vector on the identity kernel heap under
   CONF_fork (ATFORK_KH), and (b) skip any handler whose address is not owned by
   a currently-loaded ELF object (elf::object_containing_addr).

2. drivers/virtio-blk.cc: a forked backend issues a ZFS read/write; make_request
   does new blk_req(bio) on the backend thread, so the request lands in that
   backend COW arena.  The single blk completion thread (req_done->drain_queue
   in AS0) then delete req -> fork_arena::free reads a divergent arena chunk
   header (magic mismatch) -> abort.  Fix: allocate blk_req on the identity
   kernel heap under CONF_fork (same rule as the virtio-net TX net_req fix).

…L heap, libsolaris.so statics, large arrays) -> stock PostgreSQL completes recovery + reaches ready on ZFS-on-NVMe on OSv

The prior ZFS-era fork work (18f59ac3) let PostgreSQL fork its aux processes on
ZFS-on-NVMe but the startup process then hung: a LOST WAKEUP in the ZFS
synchronous-I/O completion path across the fork copy-on-write address-space
boundary.  Root cause (gdb-proven): ZFS runs as libsolaris.so loaded into the
COW-cloned application VA slot (slot 32), so ALL its state -- .data/.bss statics
(buf_hash_table, arc_anon/mru/mfu, the dbuf hash), its SPL/kmem heap objects
(zio_t.io_cv, zil_commit_waiter_t.zcw_cv, dmu_tx/dbuf/dnode/arc_buf_hdr and their
embedded locks/condvars), its 8 MB ARC/dbuf hash arrays, and the block-I/O bio --
gets a private COW copy per forked child.  A forked PG process blocks on a
primitive in its copy while ZFS kernel threads in AS0 (the virtio-blk completion
thread, txg_sync, dp_sync_taskq, the zil lwb writer) signal / dereference a
DIFFERENT physical copy: the wakeup is lost (all vCPUs idle) or AS0 faults on a
stale pointer.

Fixes (all #if CONF_fork; non-fork builds byte-identical):
 - fs/vfs/kern_physio.cc: alloc_bio() on the identity kernel heap (the block
   completion thread in AS0 reads bio->bio_caller1/bio_done to drive zio_interrupt
   -> zio_done -> cv_broadcast; the bio must be one coherent object).
 - opensolaris_kmem.c: route zfs_kmem_alloc + kmem_cache_alloc (the whole SPL/ZFS
   heap) onto the identity kernel heap -- ZFS objects are shared kernel state, not
   per-process app state.  free() dispatches by address so this is free-safe.
 - core/fork_arena.cc + exported_symbols/osv_libsolaris.so.symbols: C-linkage
   fork_kernel_heap_push/pop accessors so the libsolaris.so module can route its
   allocations to the identity heap.
 - core/mmu.cc + include/osv/mmu.hh + core/elf.cc: share libsolaris.so writable
   PT_LOAD segments (.data/.bss) verbatim (never COW) across every fork child, so
   its global ZFS state (buf_hash_table, arc_anon, dbuf hash, ...) is coherent in
   every address space.  The ELF loader registers each such segment.
 - core/mempool.cc: large allocations (>= 2 MB, mapped_malloc_large) made under
   force_kernel_heap are registered fork-shared too -- catches ZFS 8 MB ARC/dbuf
   hash arrays, which otherwise land COW in the app mmap slot.

Result: PG now completes full crash recovery (was: hung immediately after
database-system-was-shut-down) and reaches database-system-is-ready-to-accept-
connections on ZFS-on-NVMe.  A further wall remains in the ZIL lwb synchronous-
write completion (fsync -> zil_commit -> zil_commit_waiter cv_wait; lwb stuck
ISSUED) -- documented for follow-up.

Author: Greg Burd

…ntity kernel heap so forked-backend writes are not deadlocked

A ZFS taskqueue (struct taskqueue: its tq_mutex, tq_queue list heads and
tq_threads array) is SHARED kernel infrastructure: the AS0 txg_sync / zio
pipeline threads take tq_mutex and splice tasks onto tq_queue, while forked
PostgreSQL backends enqueue onto the SAME taskqueue.  subr_taskqueue.c
allocated it with plain calloc(), which std_malloc routes to the per-process
fork COW arena.  After the first fork() the arena page holding tq_mutex is
COW-write-protected in AS0, so an AS0 sync thread issuing a write ZIO
(spa_sync -> dbuf_sync_list -> zio_nowait -> zio_issue_async ->
taskq_dispatch_ent -> taskqueue_enqueue -> mtx_lock(&tq->tq_mutex)) takes a
COW WRITE fault.  vm_fault grabs the kernel vma_list_mutex for write and
deadlocks against a forked backend holding it for read across a demand fault;
txg_sync wedges and forked-backend dirty data never reaches the disk (the
virtio-blk WRITE counter never moves).

gdb evidence: tx_sync (spa_sync/dbuf_write/taskqueue_enqueue) blocked in
rwlock::wlock on a lock xadd against a mutex at 0x3000000b4fd0 (FORK-ARENA COW,
PTE writable=ro); vma_list_mutex held WRITER_LOCK, txg syncing/synced frozen
across samples.

Fix: wrap the struct taskqueue + tq_threads allocations in the identity
kernel-heap scope (fork_kernel_heap_push/pop) so the taskqueue is one physical
object shared by every address space (never COW).  Single-process OpenZFS is
unaffected (no fork, no COW).  Gated CONF_fork: conf_fork=0 is byte-identical
(the TQ_KHEAP macros expand to no-ops).

Signed-off-by: Greg Burd <greg@burd.me>

…_size after write

zfs_vop_write() (the OSv vop_write bridge) calls zfs_write(), which updates the
ZFS logical file size (zp->z_size) but NOT the OSv vnode cached size
(vp->v_size).  zfs_vop_read()/zfs_vop_cache() bound reads with vp->v_size, so a
file-extending write followed by a read of the newly written region returns
end-of-file even though the bytes are in the ARC and on disk.  PostgreSQL hits
this immediately: a backend extends a relation (writes block 0) and a later
read fails with "unexpected data beyond EOF in block 0 of relation ...", so no
committed row is visible and no workload can run.

The fix (a new OSv platform-layer patch in modules/open_zfs/patches/, NOT a
direct edit of the pinned submodule) refreshes vp->v_size from zp->z_size after
every successful regular-file write.  This is a single-process correctness bug
(independent of fork/COW); it was latent because earlier bring-up used
write-then-close-reopen or reads within the initial size, never a
write-extend-then-read on the same live vnode.

Verified: create table + insert + checkpoint + select round-trips (rows
visible), and the data survives a guest crash + reboot + pool re-import.

Signed-off-by: Greg Burd <greg@burd.me>

OSv shmctl(IPC_STAT) unconditionally returned 0 and shmat(dup(fd)) mapped
whatever unrelated file happened to occupy that fd number.  A stale SysV shmid
recorded on disk (e.g. PostgreSQL postmaster.pid after an unclean shutdown) was
therefore mistaken for a live segment across a reboot: shmget key registry is
per-boot RAM, so after a reboot the old id no longer names a shm_file, but
PGSharedMemoryIsInUse() probing that id got SHMSTATE_ANALYSIS_FAILURE and PG
refused to start with "pre-existing shared memory block (key .., ID ..) is
still in use".  This blocked PostgreSQL crash recovery on OSv.

Add shm_fd_is_segment() (dynamic_cast<mmu::shm_file*> of the fd) and reject
shmids that are not live shm segments with EINVAL in shmctl(IPC_STAT/IPC_RMID)
and shmat; PGSharedMemoryAttach then correctly concludes SHMSTATE_ENOENT and PG
proceeds with automatic recovery.  Wrong even for single-process PostgreSQL
(any restart after a crash); not fork-specific.

Verified: after a guest crash + reboot PostgreSQL runs redo ("automatic
recovery in progress") and reaches "ready to accept connections", and the
pre-crash committed rows are readable.

Signed-off-by: Greg Burd <greg@burd.me>

…ession

drivers/virtio-blk.cc: add four volatile global counters (g_blk_submitted /
g_blk_completed and the VIRTIO_BLK_T_OUT-only g_blk_wr_submitted /
g_blk_wr_completed) incremented in make_request() and drain_queue().  These are
the read-via-gdb instrument used to prove forked-backend writes reach the
block device (the WRITE counter must MOVE across insert+checkpoint).  Small,
always-on, benign.

tests/tst-fork-zfs-write.cc (+ modules/tests/Makefile, in zfs-only-tests): a
forked child writes+fsyncs a 40 KiB file on a ZFS mount (/tmp) and the parent,
after reaping the child, must read back the exact bytes at the grown size.
Reproduces both write-path walls (taskqueue COW deadlock -> hang/no-persist;
v_size beyond-EOF -> short read) on a buggy kernel and passes with the fixes.
Gated CONF_fork (SKIPs otherwise).

Signed-off-by: Greg Burd <greg@burd.me>

The OpenZFS image-populate step (create_zfs_filesystem in scripts/build and the
cpiod builder in scripts/upload_manifest.py) OOMs at 512M when building the
PG-on-ZFS image; 4G is fine on the build host.  scripts/build now honours an
optional ${zfs_builder_mem} override (default 4G).

Signed-off-by: Greg Burd <greg@burd.me>

…so an AS0 writer can wake a forked-backend reader

rwlock::rlock() registers a pending reader as a lockfree::linked_item<thread*>
that is a LOCAL on the waiting thread stack, pushed onto the shared _read_waiters
queue.  With per-child COW stacks, when a forked PostgreSQL backend blocks as a
reader its read_waiter lives at a COW-private app-stack VA (0x200000...); an AS0
writer releasing the lock walks _read_waiters in wake_pending_readers() and
dereferences that VA through AS0 page tables -> SIGSEGV (Aborted in
rwlock::wake_pending_readers+82 <- rwlock::wunlock <- taskqueue_thread_loop),
which crashed a pgbench bulk COPY at ~20%.

Mirror the existing lfmutex/condvar coherent_wait_record mechanism: when
fork_child_needs_heap_wait_record() (a forked-child caller, or any live child AS
while the AS0 parent waits), allocate the linked_item from the identity kernel
heap (fork_arena::kernel_heap_scope) so its VA is coherent in every address
space; the waiter frees it after waking.  Default OSv / AS0-only keeps the
zero-overhead on-stack fast path.  Gated CONF_fork; conf_fork=0 byte-identical.

Verified: pgbench -i -s 50 (5,000,000-row bulk COPY) now completes 100% where it
previously aborted at 20%.

Signed-off-by: Greg Burd <greg@burd.me>

…y kernel heap so a forked backend's rename survives cross-AS drele()

dentry_alloc() already wraps its dentry + d_path allocations in a
fork_arena::kernel_heap_scope so shared dentry-cache infrastructure stays off
the per-address-space COW fork arena.  dentry_move() (the rename path) did NOT:
its dp->d_path = strdup(path) ran unguarded, so a rename issued by a forked
PostgreSQL backend allocated the new d_path in that backend's COW-private
fork arena.

When the last reference to that dentry later dropped in a DIFFERENT address
space (AS0 postmaster or a sibling backend), drele() -> free(dp->d_path)
routed the pointer to fork_arena::free(), which read the chunk header at that
arena VA through the freeing AS's page tables.  After the COW divergence that
VA holds a different physical page there, so the header magic mismatched and
recover() aborted:

  Assertion failed: h->magic == chunk_magic (core/fork_arena.cc: recover: 257)
  cloudius-systems#3 fork_arena::free
  cloudius-systems#4 drele (dp=...) at fs/vfs/vfs_dentry.cc:229   <- free(dp->d_path)
  cloudius-systems#8 vfs_file::close  <- close(fd)

PostgreSQL triggers this during a checkpoint that recycles WAL segments
(rename of a pg_wal file), which is exactly the "N recycled" checkpoint that
crashed a forked backend on the RAID-Z pool.

Fix: wrap dentry_move()s d_path strdup in the same kernel_heap_scope as

… AS0 so the postmaster launches parallel workers (parallel-query hang)

A forked PostgreSQL backend that signals the postmaster used kill(pid, sig)
with pid == OSV_PID (2) -- the postmaster is the top-level app / AS0, whose
getpid() is OSV_PID.  The CONF_fork kill() path resolved a cross-process target
via osv::fork::as_for_pid(pid) and returned ESRCH when that was null.  AS0 is
never entered into g_pid_as (only fork children are register_pid()'d), so
as_for_pid(OSV_PID) == nullptr and kill(OSV_PID, sig) from a child returned
ESRCH -- the signal was dropped.

That is the lost wakeup behind the parallel-query hang.  A parallel-query
leader (itself a forked backend) launches workers via
RegisterDynamicBackgroundWorker, which sets a shared request slot and then does
kill(PostmasterPid, SIGUSR1) to make the postmaster fork the workers.  With the
signal dropped (ESRCH), the postmaster -- parked in its ServerLoop
WaitEventSetWait/epoll_wait -- never sees the request, never forks a worker
(debug1 logging showed ZERO "starting background worker" lines), and the leader
waits forever in WaitForParallelWorkersToAttach.  Block I/O was fully drained;
this is a pure coordination lost-wakeup, same cross-AS/per-fork-child-pid
routing class as the shipped SIGCHLD/SIGURG-latch fixes.  (It also explains the
noted SIGHUP-to-postmaster ESRCH from pg_reload_conf().)

Fix (libc/signal.cc, #if CONF_fork): exclude OSV_PID from the cross-process
target resolution -- treat it like the self/broadcast case so target_as stays
nullptr and the handler runs in AS0 (mmu::kernel_address_space()), which is
exactly where the postmaster's handler must run to poke its own latch/self-pipe
and wake its ServerLoop.  Never returns ESRCH for the top-level app.  The
later (pid == OSV_PID) wake_up_signal_waiters()/handler-in-AS0 logic is
unchanged.  conf_fork=0 does not compile this block (byte-identical).

Validated: with max_parallel_workers_per_gather=2, a parallel Seq Scan Gather
`select count(*) from pgbench_accounts` returns 10000000 (== the non-parallel
reference), and EXPLAIN ANALYZE shows "Workers Planned: 2 / Workers Launched: 2"
with both workers processing rows.  Previously: 0 workers launched, hang.

Signed-off-by: Greg Burd <greg@burd.me>

…_simple + zfs_space on OSv so ZIL replay works (sync=standard durability)

The OSv OpenZFS platform layer left zfs_write_simple() and zfs_space() as
ENOTSUP stubs.  Both are on the ZIL replay path that runs on pool import after
a crash when a dataset has sync=standard (ZIL active, here on a SLOG vdev):
  zfs_replay_write()    (TX_WRITE, txtype 9)  -> zfs_write_simple()
  zfs_replay_truncate() (TX_TRUNCATE, txtype 10) -> zfs_space(F_FREESP)

Consequence on the RAID-Z (4xEBS raidz1 + NVMe SLOG + L2ARC): a kill -9 mid-txg
with sync=standard, then re-import, logged
  "ZFS replay transaction error 95 [ENOTSUP], dataset pgdata, txtype 9/10"
and dropped every logged intent-log record (ZIL gave no durability); an earlier
build variant instead SPL-PANICked in abd_alloc_linear
(VERIFY3U(size <= SPA_MAXBLOCKSIZE)) on a log record it could not replay.
Either way sync=standard was not usably crash-recoverable through the ZIL on OSv
-- forcing sync=disabled and losing the SLOG's purpose.

Patch 0029 implements both the OSv way, reusing primitives that already work
here (so this is a single-process correctness fix, no fork/COW dependency,
correct for every config -- deliberately NOT CONF_fork-gated):
  * zfs_write_simple: one-iovec struct uio over the kernel buffer +
    zfs_write(zp, &zuio, O_SYNC, NULL) (the zfs_vop_write shape / FreeBSD's
    vn_rdwr(UIO_SYSSPACE, IO_SYNC)); refresh vp->v_size from zp->z_size.
  * zfs_space: F_FREESP wrapper delegating to the already-present
    zfs_freesp() (zfs_znode_os.c), identical to FreeBSD's zfs_space.

As an upstream-openzfs source change this is a modules/open_zfs/patches/ entry
(git-apply'd to the pinned submodule at build time), NOT a submodule edit.

Validated on the RAID-Z: sync=standard, insert marker + checkpoint + settle,
kill -9, reboot -> NO replay error, NO panic, ZIL TX_WRITE/TX_TRUNCATE replay,
PG crash recovery completes, and the row survives crash+reboot.

Signed-off-by: Greg Burd <greg@burd.me>

…bling fork backends (PG shared_buffers catalog-read-zero)

Stock PostgreSQL (shared_memory_type=mmap, the default) puts shared_buffers
in one big anonymous MAP_SHARED segment the postmaster creates BEFORE forking
backends.  Only a few of its pages are touched pre-fork; the system-catalog
index/relcache pages are first-faulted by an individual backend AFTER its
fork.  On OSv this produced the multi-backend catalog-read-zero wall: backend
cloudius-systems#1 reads a catalog-index page fine, but sibling backends cloudius-systems#2+ (zero writes in
between) read the SAME shared page as ZEROS ("pg_authid_rolname_index contains
unexpected zero page", "cache lookup failed"), so HammerDB can't even
authenticate vuser cloudius-systems#2.  Hypervisor-independent (KVM and Firecracker identical).

Root cause (core/mmu.cc): an anon MAP_SHARED page not present in the parent's
page table at fork time is cloned as an EMPTY child PTE (clone_pt_level0 can
only share what is already present).  When a sibling backend later faults that
VA, initialized_anonymous_page_provider::map() calls memory::alloc_page() and
installs a FRESH, PRIVATE, ZERO page in that backend's AS only.  Every AS that
first-touches the page post-fork gets a different private zero page -- they
never converge on the ONE physical page MAP_SHARED promises.

Fix: back an anon MAP_SHARED vma with a shared_anon_page_provider that keys a
process-global registry (identity kernel heap, like shm_file::_pages) on the
absolute page VA.  An anon MAP_SHARED region lives at the SAME VA in every fork
AS (clone_address_space keeps identical VAs), so the VA uniquely identifies the
shared page: the first AS to fault it allocates+records it; every later AS
(sibling backend or parent) that faults the same VA maps THAT frame.  All
siblings converge.  The provider tags the PTE pte_shared so per-AS teardown
(free_child_pt_level0) does not free the jointly-owned frame -- without this,
the first backend to exit freed the shared page and later siblings read
freed/zeroed memory (the "second backend reads zeros" symptom).  A searched
(addr==NULL) mapping is relocated by allocate() after construction, so the
provider base is re-keyed via anon_vma::update_shared_base() once the real VA
is known.

Regression test tests/tst-fork-shared-catalog.cc mirrors PG exactly: parent
creates an anon MAP_SHARED region, forks THREE siblings; child A writes a page,
siblings B and C (and the parent) must read A's data, not zeros.  Reproduces
the zero-read on HEAD; PASSES with the fix at -smp 2 and -smp 4 (5/5 runs, real
concurrency, not gdb-serialized).  Full fork suite still 0 failures
(tst-fork/cow/deep/child-mmap/file-mmap/posix-shm/shared-write/shared-race/
serial/sigchld-latch/arena-freelist/irq-cow/preempt/socket/conn-socket/
timer-park-stress).  All #if CONF_fork; non-fork byte-identical.

Author: Greg Burd <greg@burd.me>

…kq quiescence

OSv taskq_wait() enqueued a single barrier task and drained only that
barrier. With an 8-worker system_taskq a free worker runs the barrier
while the other 7 are still mid dnode_sync/dbuf_sync, so taskq_wait
returned early and the syncing thread raced the still-running workers ->
dbuf/dirty-record/refcount corruption (arc_write_done VERIFY3S refcount
underflow + lfmutex owner-mismatch panics). ZFS documents this contract
(dmu_objset.c PORTING note). Add taskqueue_drain_all() which waits until
the queue is empty AND no worker is active (illumos semantics), and wire
taskq_wait() to it. conf_fork unaffected.

…tity linear map + heap-allocate reclaimer waiter node for fork children

Under CONF_fork, ZFS large allocations (>= huge_page, or the non-contiguous
fallback) made under fork_arena::kernel_heap_scope (force_kernel_heap) went to
map_anon(), which lands in the CURRENT thread address space app mmap slot
(VA 0x2000..). A forked PostgreSQL backend making such an allocation (e.g. an
8k zio_buf / dbuf db_data) put the VA only in that child page tables; AS0
txg_sync / zio-completion threads and sibling backends faulted outside
application on db_data. Force force_kernel_heap large allocations onto
free_page_ranges (linear map, phys_mem 0x4000.., a kernel PML4 slot mapped
verbatim in every AS) and never fall back to map_anon. Verified buffers move
0x2000 -> 0x4000.

Companion: reclaimer_waiters::wait() wait_node is a stack local dereferenced by
the AS0 reclaimer; for a fork-child waiter its COW-private stack VA GPFd
wake_waiters(). Heap-allocate the node for fork children (identity, coherent),
same pattern as coherent_wait_record. Non-fork path byte-identical.

Signed-off-by: Greg Burd <greg@burd.me>

…-swap asm (no post-swap %rbp stack read)

The fork COW context switch (switch_as branch) re-tested switch_as and reloaded
the FPU control words (fpucw/mxcsr) in C++ AFTER the inline asm swapped rsp/rbp
AND CR3 to the incoming fork child.  The compiler spills switch_as and the
fpucw/mxcsr locals to %rbp-relative stack slots, and %rbp at that point already
points at the incoming child stack under the just-loaded child CR3.  So the
post-swap code

    mov -0x50(%rbp),%rdx ; cmp %rdx,-0x48(%rbp)      (the switch_as re-test)
    movzwl -0x52(%rbp),%eax ; mov %ax,-0x34(%rbp)    (the else-branch reload)

reads and writes the child COW stack in an irq-off, non-preemptable window.  A
COW write-fault there trips assert(sched::preemptable()) at arch/x64/mmu.cc:38,
and a mis-read switch_as takes the wrong branch and can ldmxcsr a garbage MXCSR
(reserved bits set) -> #GP, so the resumed child never runs again (a forked
backend parks in switch_to and is never rescheduled).  The prior FPU fix
(1ae2602f9) hardened the reloaded VALUES for the switch_as==true path but left
the branch decision and the else-branch operands as %rbp-relative reads, still
resolved through the incoming child AS.  Dormant when the surrounding code
compiles exactly like c8f9c82b; exposed once code layout shifts the fork timing.

Fix: emit emms + fldcw + ldmxcsr from kernel-identity-mapped .rodata statics
INSIDE the switch_as asm, right at the 1: resume label, then return.  RIP-relative
.rodata is identically mapped in every address space (never COW), so nothing after
the CR3 swap touches a %rbp-relative (stack) slot: no COW fault, no mis-read.  The
canonical control words are correct for every OSv thread (issue cloudius-systems#1020; switch_to
does no fxsave/fxrstor, so no per-thread FPU state is preserved across a switch
regardless), so this is semantically identical to the old fldcw(fpucw)/ldmxcsr(mxcsr)
for the AS-crossing case.  The shared post-asm reload now runs only for the
non-crossing (same-AS) path, whose %rbp is the resumed threads own coherent stack.

…e self-pipe coherent

Two coupled CONF_fork fixes so a forked PostgreSQL backend's own latch and
ProcSignalBarrier self-signals reach it, without breaking the child-exit
notification.

kill(): a fork backend's OWN SIGURG (latch) and SIGUSR1 (ProcSignalBarrier)
self-signal must run its handler in the caller's own copy-on-write address
space, so the handler sets the pending flag / pokes the self-pipe in the copy
the backend actually reads. Restrict the self-routing to SIGURG/SIGUSR1: the
child-exit SIGCHLD that fork emulation raises with kill(getpid(), SIGCHLD) from
a dying child must NOT be diverted, or the parent's reaper runs against the
child's torn-down mapping and faults, and the parent never reaps the exited
child (it then never finishes startup). as_for_pid() also gates to a live
registered child, so a child mid-teardown falls back to the top-level table.

pipe2(): allocate the pipe_buffer and pipe_file on the identity kernel heap so
the self-pipe backing a latch is coherent across every fork address space (a
cross-backend wakeup writes the pipe from another address space while the
waiting backend polls the same pipe). Same rule as the epoll and file
containers.

Both are inside #if CONF_fork, so a non-fork build is byte-identical.

Validated: with the routing present, boot-to-ready 10/10 on a fresh recordsize=8k
pool; CREATE DATABASE + DROP DATABASE completes 5/5 (was an indefinite wait for
the emitter's own ProcSignalBarrier); pgbench -c8 0 failed transactions.

Two independent bugs prevented building the fork-enabled kernel
(conf_fork=1) on GCC 11.x, and also broke a clean conf_fork=0 build on
both GCC 11 and GCC 13:

1. core/mmu.cc defined anon_vma::~anon_vma() unconditionally, gating only
   the body with #if CONF_fork, while include/osv/mmu.hh declares the
   destructor only under #if CONF_fork. When CONF_fork is 0 the class
   does not declare the destructor but the .cc still defines one, which
   is an ISO C++ violation ("definition of implicitly-declared
   destructor") rejected by both GCC 11 and GCC 13. Move the #if CONF_fork
   to wrap the whole destructor definition so it matches the declaration.

2. Makefile auto-generates the kernel configuration with
   $(shell make -f conf/Makefile -j1 config), which did not forward
   conf_fork. GNU make does not propagate a command-line variable
   override into a $(shell ...)-invoked sub-make, so the Kconfig option
   (def_bool driven by the conf_fork env var) always resolved to
   CONF_fork=0. A conf_fork=1 build therefore produced CONF_fork=0 and
   silently failed to compile the fork sources. Forward conf_fork to the
   config sub-make.

Both changes are gated on CONF_fork / conf_fork. conf_fork=0 output is
unchanged: at CONF_fork=1 the mmu.cc change is a preprocessor no-op
(the -g0 core/mmu.o is byte-identical), and forwarding an empty conf_fork
yields the same generated config as before.

Verified: fork+ZFS kernel compiles and links clean and the resulting
image boots (ZFS root mounted) on both GCC 11.5 and GCC 13.3.

Signed-off-by: Greg Burd <greg@burd.me>

…o a forked backend does not fault in elf::resolve_pltgot

OSv resolves a dynamic object's PLT/GOT lazily: the first call to an imported
function traps into elf::object::resolve_pltgot(), which looks the symbol up
and writes the resolved address into the object's .got.plt slot. Under
CONF_fork a forked child runs in a copy-on-write clone of the parent address
space, and two facets of the lazy resolver are not coherent across that clone,
which surfaces as a NULL dereference inside elf::resolve_pltgot in a forked
child (e.g. a PostgreSQL backend under a heavy stored-procedure workload).

Root cause and fix, both CONF_fork-gated (conf_fork=0 emits identical code):

1) COW-stale GOT. The writable file-backed segment that holds .data/.got/.got.plt
   is demand-paged. The dynamic linker RELOCATES that GOT in memory at load
   (object::relocate / relocate_pltgot rewrite each slot by the object base and
   install the resolver trampoline pointer and the object pointer in pltgot[1]).
   Those writes diverge the page from its on-disk bytes. If such a relocated
   page is not resident when clone_address_space() clones the parent, the child
   inherits an empty leaf PTE and its first access re-reads the ORIGINAL,
   unrelocated bytes from the file (pltgot[1]=0, raw jump slots). The child's
   lazy resolver then dereferences a NULL elf::object and faults. Fix:
   mmap_populate writable file-backed segments at load so every relocated page
   is resident in the parent; clone_address_space then COW-shares the relocated
   pages and the child always reads the correct GOT. OSv does not swap and a
   written file-private page becomes anonymous, so a populated page stays
   resident through fork.

2) Symbol-set node in the child COW arena. When a forked child is first to
   resolve a cross-object slot, resolve_pltgot() inserts into the shared
   elf::object's _used_by_resolve_plt_got set. Without care the set node is
   allocated from the child's private COW arena while the set lives on the
   shared identity heap; a sibling or the top-level address space later walking
   that set dereferences a node mapped only in the child's (possibly already
   reclaimed) address space. Fix: allocate the set node under a
   kernel_heap_scope so it lands on the identity heap, coherent across every
   fork address space.

Adds tests/tst-fork-pltgot.cc: a forked child is first to resolve a batch of
libc/libm PLT symbols the parent never called, plus a parent-and-reaped-children
cross-object stress of the shared symbol set. Passes at -smp 4 and -smp 8;
boot-to-ready 10/10 unaffected.

…orked child runs the lazy resolver (fixes the fork symbol-lookup wall)

Under CONF_fork, force bind_now=true in object::relocate_pltgot() so every
PLT jump slot is resolved in AS0 at load time, before any fork().

Lazy PLT binding is not fork-coherent.  A forked child running in its own
copy-on-write address space that is the FIRST to hit an unresolved PLT slot
must run the resolver (elf::object::resolve_pltgot -> object::symbol ->
program::lookup) inside that child.  Two distinct facets fault there:

  1. GOT-slot relocation (fixed in 8d775cc): a demand-paged, relocation
     -dirtied GOT page not resident at fork is re-read UNRELOCATED from the
     backing file in the child, so the resolver dereferences a NULL
     elf::object.

  2. Symbol lookup (this change): under heavy fork load (a workload that
     forks many worker processes which each first-call library functions the
     parent never touched), a child's lazy resolver RUNS but the dynamic
     symbol LOOKUP fails for a symbol that IS exported by the kernel and has a
     normal undefined ref in the app (e.g. sem_wait, preadv, pwritev -- POSIX
     semaphores and vectored I/O), the "symbol not found" wall.

Resolving every slot up front in AS0 removes BOTH by construction: no forked
child ever enters the resolver.  The resolution also WRITES each GOT slot, so
its page becomes anonymous and resident and clone_address_space() COW-shares
the RELOCATED contents.  bind_now runs in relocate_pltgot() before
fix_permissions() write-protects the RELRO segment, so the slot writes never
hit a read-only page (the failure mode an earlier post-fork eager-bind attempt
hit).

This supersedes the mmap_populate-of-writable-segments half of 8d775cc (the
bind-now writes already pin the GOT resident) and is complementary to its
identity-heap _used_by_resolve_plt_got insert, which now only matters for
objects dlopen'd AFTER the first fork.

Evidence (conf_fork=1, x64): a temporary trace in resolve_pltgot counted
forked-child lazy resolutions across three fork tests: without bind-now
102 / 263 / 681 child resolutions; with bind-now 0 / 0 / 0.  New tests
tst-fork-symlookup (heavy concurrent fork, wide symbol set incl.
sem_wait/preadv/pwritev) and tst-fork-vislookup (PLT-linked dependency
resolved first in forked child + a pthread) pass at -smp4 and -smp8, as does
tst-fork-pltgot.  Boot 10/10.  conf_fork=0 elf.o is byte-identical to
8d775cc modulo 11 __LINE__ immediates; zero real code differences.